1. The Field of the Invention
The present invention relates to measuring the reliability of software modules. More specifically, the present invention relates to systems, methods, and computer program products for generating language independent state tables usable when measuring the reliability of different versions of a software module, automatically verifying that a version of a software module performed an action as intended, and performing customized testing functions associated with a version of a software module.
2. Background and Relevant Art
Software vendors often release software products, such as user interfaces and application programs, in a variety of written languages and for use on a variety of computer platforms. This is in large part due to the number of differently configured computer systems operating worldwide in different countries. If a software product is released in a variety of languages and/or for use with a variety of computer platforms, finding and predicting the severity of defects (or xe2x80x9cbugsxe2x80x9d) in the software product becomes very important.
Many consumers install a wide variety of software products on any given computer system. These software products must interact in a manner that does not corrupt data or cause the computer system to crash. To decrease the chances of such problems occurring, it is beneficial to measure the reliability of software products under a wide array of different circumstances. Thus, versions of software products in different written languages may often be tested for use on a variety of computer platforms.
One conventional approach to measuring software reliability is to hire and train skilled software testers to systematically exercise a software product to find bugs. However, this approach to testing is highly labor intensive, and it is often very expensive to hire and train testers. Also, there is also some likelihood that a tester may miss a critical function of a software product or an interaction between several software products, which may allow a defect to go undiscovered and thus uncorrected.
Another conventional approach to measuring software reliability is to send a partially complete software product out to a large population of users to test the product. However, it is often expensive to create a software product, even one with partial functionality, and distribute it to a large number of users. Additionally, competitors of a software vendor may obtain a pre-release version of the product thus reducing the product""s competitive advantage. There are also logistical problems associated with obtaining any useful information about a software product when test reports are received from a large number of users. Furthermore, similar to skilled software testers, users of a pre-release version of a software product may miss critical functions.
To lessen the chance for human error when testing the reliability of a software product, many vendors began creating batteries of automated software tests. These tests simulate actions that a user may typically cause a software product to perform. However, conventional automated test batteries have some inherent problems: batteries of tests are expensive to create and maintain; they are often self-contained and do not interact with other testing programs; and they often test only a small portion of the software modules included in a software product. Since different batteries of tests are often self-contained and incompatible, batteries for individual software products must be individually coded. This is time consuming, may require substantially technical expertise on the part of the tester, and increases the chance of introducing human error into the testing process.
More recent approaches to testing software reliability use a modular approach where different testing modules perform different functions of the testing process. A state table editor may be used to create databases called state tables, which include object values and actions associated with particular states of a target software product. State tables may be created by entering some information manually and detecting and retaining other information from an operating system. Detected and retained information may include operating system values, such as the values associated with buttons, scrollbars and dialog boxes, and user actions, such as mouse clicks and keyboard typing, into a database.
A state table contains various entries, each entry representing a state that the software module may potentially enter. For each potential state, there is a list of actions that are available once that state is entered. For each possible action, the state table may include a pointer to the state that the software module should enter if that action is performed. For each possible action, the state table may store a likelihood that a user may perform that action given that state.
To test a software module included in a software product, a state table runner xe2x80x9creadsxe2x80x9d a state table entry corresponding to a particular state and causes a software module to perform an action associated with that state, based on a likelihood percentage assigned to the action. For example, if a state table likelihood for a particular action equals 10%, the state table runner will perform this action one out of every ten times that it enters the particular state. Thus, unlike testing methods employing human testers, the likelihood of executing functions with a higher or lower probability than they would actually occur is reduced. The state table runner is also capable of verifying that a correct result occurred when an action is performed and of detecting the level of failure of an action. As the state table runner performs the actions included in the state table, it may record each action it performs and the result of performing the action
While conventional modular approaches provide benefits when compared to human testing or hard-coded automated testing methods, they also have certain problems. For a reliability test to provide meaningful results, a state table associated with a software module must be properly configured before a state table runner reads the state table. State table runners rely heavily on text string values included in tested software modules, such as window titles and dialog box contents, when determining if an action performed as intended. In a state table, these text strings are often hard-coded, which may require an individual to manually enter the text strings.
The problem is significantly compounded when versions of a software application are created in different languages. If an English, German, and French version of a software product is to be released, text strings associated with each language may need to be manually entered into separate state tables, where a state table is dedicated for each language version. Additionally, the need to manually edit an existing state table may arise when developers of a software product change text strings included in the product. Manually entering or editing text string values in a state table increases the operational complexity associated with testing, consumes time, and increases the chance that human error will be introduced into the testing process.
Another problem is that different versions of a software product may exist for different computer platforms. That is, a version of a software product designed for use on mobile devices may operate differently on a cell phone than on a personal digital assistant. Thus, when using conventional modular approaches to test a software product, a separate state table may need to be created for each computer platform compatible with a version of the product. Furthermore, if customized testing functions are to be performed such as actions not included in the functionality of a generic state table runner, these functions may need to be coded into different versions of a state table runner specifically designed for each computer platform. Coding a state table runner is time consuming, increases the chance of introducing human error into the testing process, and may require significant technical expertise.
Therefore, what are desired are systems, methods, and computer program products for automatically testing the reliability of software modules in a language independent manner where verifications of a software module""s state and simulating actions performed by a software module may be more easily customized.
The principles of the present invention provide for automatically testing the reliability of software modules in a language independent manner. A state table editor uses standardized pointers, which are common to a number of versions of a software module, to associate states included in a state table to objects included in a tested software module. Additionally, customized verifications and actions may easily be added when testing versions of software modules associated with different user interfaces and computer system platforms. Loadable program modules, which receive standardized data via common headers, may be used to facilitate these customized verifications and actions.
Language independence is facilitated by performing actions and verifications included in a language independent state table. To generate a language independent state table, a resource file is accessed to locate pointers that are standardized across different versions of a software module. In one instance, existing IDN files may be accessed to locate keys associated with the objects of a software module. The standardized pointers may then be received by a state table editor, which includes the standardized pointers in a state table associated with the software module. Since the keys reference objects, instead of the values of objects, no modifications need be made to a state table if a developer changes a portion of an object, because the key still references the correct object. As a result, the same state table may be used for a number of versions of a software module, each of which outputs similar objects, but in a different written language. The efficiency in measuring the reliability of a software module is greatly increased when one state table may be used to facilitate testing for a number of software module versions.
In operation, methods of the present invention may verify that an action performed by a software module produced the intended result. In one embodiment, the standardized pointer is used to retrieve parameters representative of an anticipated state. An anticipated state is a state that a software module should transition into when an action is performed. The retrieved parameters may include strings of text, such as the title or contents of a dialog box, that are included in a resource file.
In a separate operation, an action may be performed that causes a software module to transition to an actual state. An actual state is a state a software module transitions into as a result of performing an action. This may include a state table runner evaluating the various actions available given its current state, and then performing an action according to the likelihood assigned to that action. Such actions may include, for example, clicking a mouse button at a particular location, and/or pressing keys on a keyboard.
It is then determined that the actual state and the anticipated state have one or more representative parameters that are at least a partial match. If information representative of an actual state and an anticipated state satisfy a predetermined level of commonality, this may verify that the action preformed as intended. Depending on the strictness of the rules involved when comparing information or parameters, this may be determined in a variety of ways. If one or more parameters representative of the states partially match, this may be enough to verify that an action performed as intended. In some cases, only a complete match of parameters may result in verification. In other cases, a partial or complete match of different representative parameters may result in verification. Comparisons to determine if representative parameters are a partial or complete match may be performed on visual objects of a display, such as text associated with the title or contents of a dialog box.
Using standardized pointers to facilitate retrieval of information from existing resource files and comparing information representative of an anticipated state and an actual state significantly reduces the chance of human error due to entry of erroneous data and significantly reduces the operational complexity associated with testing.
Customized actions and verifications for specific software modules are facilitated through the use of loadable program modules, which include functionality not otherwise available during the testing process. Loadable program modules are created with common headers that, when called, receive standardized data about a state of the software module. Although a loadable program module receives standardized data, the programming instructions contained therein may be coded to perform customized functions for different versions of a software module or computer platform. As a result, adding customized functionality does not require modifying the code of an existing state table runner. Groups of loadable modules common to a software module or computer platform may be included in libraries, such as dynamic link libraries (xe2x80x9cDLLsxe2x80x9d).
In operation, performance of a selected function is requested. The request may be caused by a software module or computer system platform. As a result of the request, at least one customized function is loaded from a source external to a state table runner. Loaded customized functions may be included in a DLL that is referenced in a configuration file associated with a state table runner. Customized functions may also be specific to a software module or computer platform.
Information representative of a state of a software module is then sent to the selected function. This information may be sent via a function header that is common to a number of software modules or computer platforms. The instructions of the selected function are then processed. Processing the instructions of the selected function may cause performance of an action or verification that was customized for a particular software module or computer platform.
Thus, previously created loadable program modules may be used to add customizable actions and verifications when testing a software module. This reduces the level of technical expertise needed to perform tests and reduces the chance that human error will be entered into the testing process.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The features and advantages of the invention may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth hereinafter.